The Ops Community ⚙️: seenu v

Open Mainframe project and Open Source

seenu v — Sat, 28 May 2022 17:48:19 +0000

About open mainframe work

Open mainframe project which really focuses on the opensource ecosystem. On mainframe particularly focuses on the projects that are really interconnecting the entire enterprise. Enterprise is very hybrid and heterogeneous in their IoT infrastructure at least what they see in forward-looking enterprises. So the projects and efforts they host are meant to be sort of the interconnecting piece of the mainframe within the world.

Linux Foundation

The Linux foundation host over 400 open-source projects. Pretty much every horizontal and vertical out there and a lot of it really focus is that concept of innovation built on sort of that decentralized trust. The projects that are hosted here are all focused on the area that they all come here because they need that neutral area to collaborate.

Why would students do contribute?

The basic thing is you’re just learning these technologies and the best way to always start in any open-source project is to show up to contribute to starting playing with code, to start adding some features. Even just correcting little errors adding some tests (or) documentation. This is getting you to sort of familiar with a bit of the code and at the same time you’re getting familiar with the community. When people are really getting involved in open-source projects there’s the one half of participating. You use this code in such a way and you want to help contribute some of the use cases the code behind the use cases that you drove and bring those back into the open-source. And really amazing thing is to build your resume. You sort have a unique opportunity to be working with leaders in industry and leaders and leading companies and finally getting hired!

Difference between open-source and Internship...

Open-source community stand point is the output at your work is open-source, so people can see exactly what you’ve done the impacts that you have had and you have a true portfolio building begin.

Internship shows you can work with people in an office setting and have an idea of how offices work, a good recommendation from the company you intern for can show that you have programming skills and other important soft skills. Internships also have the potential to lead to a full-time position within the company. Contributing to open-source projects really just shows you can program. Basically working on open source projects is similar to community service, its helpful and can make you feel good, but it’s not the most efficient/best way to improve a resume.

What is mainframe work..?

It is basically a computer system that is designed really around four key design principles:
(a)Security, (b)Stability, (c) Performance, (d) Scalability.
And we see that showcased in a number of key industry areas where you need all of those aspects of your computing system turned up to an algorithm.

Example: Credit card operation, booking tickets, etc.

Open Mainframe Project

It was founded six years ago as of 2021, was founded on the case that open-source is such a critical part of where the mainframe community is going, really to be fair a lot of open-source actually came from the mainframe world. When you go back to 1955 with an organization called “share” that is really the root of where open-source came from for lack of return I mean it’s words roots computer operators f the time had this new Ibm system and they needed to know how to use it, they needed some help in being able to collaborate and so they came together and they shared insight, and they shared code! , they shared experiences, they shared insight and that has built over decades and decades and decades….. So this was really an effort here of how do we get the larger mainframe community which is also very passionate, very interconnected and also looking forward to long term sustainability of collaborating in open source. And it really started with work around the Linux usage on the s390x mainframe platform and in time it’s grown also to open-source projects. Open mainframe projectsits in there and fills the piece and it hosts the project that is very key to making that connection and as a way, it’s really bringing the mainframe into the mainstream.

Z-architecture

Basically, it is similar to the s390x mainframe platform. It is the architecture of these boxes of how they’re put together. And the degree of processing power, degree of redundancy built into the box, degree of additional hardware bits really make these boxes the most performant resilient scalable.

How beginners can contribute..?

All the projects are completely transparent open projects that anybody can get involved with not only source code available under OSI licenses but getting involved in the community very easy and What I recommend is people get on the mailing list, get into the slack channel because there are tons of communications that are happing on both sides you can start to get a feel for what the community is about what the current topics are things of that nature. Many of these projects will also maintain good first issue, tagged issues so you can go there and check what those look like and maybe use those for some low-hanging fruit that you can get involved with. Each of them has a few different tech stacks to take some of the advantages.

You can find all Open mainframe Projects on Github. Even beginners can contribute.

They are also providing mentorship program:you can apply

Cloud computing

seenu v — Fri, 27 May 2022 15:02:13 +0000

Before cloud

Let's say you have started a startup, and are focusing on building the platform. So, these are some of the things you have taken into consideration to make it success.

1. Backup system
2. Servers
3. Database
4. Network
5. Version control

So, you need a dedicated safe room and some computers to do all this stuff. So you build your own infrastructure (on-premise).

Problem with on-premise are:

Server cost
Energy cost
Wastage of manpower (You have to have a dedicated team to focus on this)
Work force
Os cost
Backup maintenance
Scaling up and down
Everytime when we build a new business app, we end up buying a new physical server.

So maintaining this data centres is not an easy task.

Here comes the picture of cloud computing

What is a Cloud?

The cloud refers to servers that are accessed over the Internet, and the software and databases that run on those servers. Cloud servers are located in data centers all over the world.

Cloud computing is the on-demand delivery of IT resources over the internet with pay-as-you-go( if you use less pay less,if you use more pay more) pricing.

Instead of buying, owning, maintaining the physical data centres and servers, you can access technology services, such as computing power, storage, and databases, on an as-needed basis from a cloud provider and just focus on building apps.

Currently AWS, Microsoft Azure, GCP are tier1 cloud providers.

Benefits of cloud computing :

Reduce cost

Cloud reduces both capex and opex
Organizations no longer have to spend huge amounts of money on physical servers and so on…

Pay as you go

You'll pay for what you use.

Scalability

Scale up, scale down, never run out of resources.

Accessibility

Accessed from virtually anywhere at any time on any machine

Business continuity

Any crisis do not result in data loss

Automatic updates
Increased collaboration
Self service

Some of the risk are:

Loss of cloud data and services
Data security
Compliance and legal risks
Cost concerns

Public Cloud

Public Cloud provides a shared platform that is accessible to the general public through an Internet connection.

Physical server, storage, networking etc.., are procured and owned by the service providers. So, no setup and maintenance worries. Anyone can use public Cloud, individuals or organisations.

Manage the cloud services and resources using cloud provider web portal
Multi-tenancy: Multiple organisations share cloud resources
Pay as you go

Benefits:

No upfront capex, pay as you go
No maintenance
High scalable
High reliable
Backup and disaster recovery solutions

Limitations:

Low visibility and control compliance and legal risks cost concerns.

Private Cloud

Located on-premise or can be hosted by a third party service provider. Private to a specific organization. Easy to customize a private cloud . Used by government and financial agencies.

Benefits:

Better security and control
Legal compliance
High regulated business

Limitations:

Limited scalability
Huge initial capex
Limited access

Hybrid cloud:

Private Cloud + Public Cloud

Benefits:

Better control
Cost effective
App continue to run in your own private cloud.
Burst through the public Cloud.

Limitations:

Low visibility and control
Additional complexity
Compliance and legal risks
Cost concerns

Types of cloud computing

Infrastructure as a service (IaaS)

Benefits:

Reduce financial risk
Deployment speed
Geographical advantages
Unlimited scalability

Examples of IaaS are:

AWS- Amazon Web Services
Microsoft Azure
Google Cloud
Digital Ocean
Alibaba Cloud

Platform as a service (Paas)

Benefits:

Reduce development time
Support global team
Develop for multiplatforms
Affordability

Examples of PaaS are:

Heroku
Elastic Beanstalk from AWS
Engine Yard
Open Shift from RedHat

Software as a service (SaaS)

Benefits:

Very easy to get started with
Accessibility
Automatic updates
Flexible usage-based pricing
Reduce financial risks
Affordability

Examples of Saas are:

Netflix
Prime
Gmail
Dropbox

DataOps!

seenu v — Thu, 26 May 2022 15:50:58 +0000

Introduction to DataOps

What is DataOps?

DataOps is the orchestration of people, processes, and technology to deliver trusted, business-ready data to data citizens, operations, and applications throughout the data lifecycle. With properly governed data, businesses can comply with complex regulations, data privacy and ensure artificial intelligence (AI) model accuracy by monitoring data quality.

This is the definition available on the web. It’s very complex for beginners, isn't it? So to put it in a brief:

DataOps brings together data scientists, analysts, developers, and operations to work on the entire product/service lifecycle, from the design stage to production support.
Sounds like the DevOps definition...But! the goal of DataOps is to make data analytics more efficient. To do so, DataOps adopts Agile Development principles, thereby improving the efficiency and effectiveness of the data teams and users.
DataOps combines principles from DevOps, Agile Development, and Lean manufacturing. Think of it as a water pipeline; your goal is to keep the water flowing in spite of all the plumbing work you carry out.

And here comes two new concepts/topics Lean Manufacturing and Agile Development. Let's see one by one:

What is Lean Manufacturing?

Lean manufacturing is a production process based on an ideology of maximising productivity while simultaneously minimising waste within a manufacturing operation. The lean principle sees waste is anything that doesn’t add value that the customers are willing to pay for.
Manufacturing happens in pipelines—raw materials flow through various manufacturing workstations to be transformed into finished goods.

What is Agile?

Let us say Mr. X is a developer. So, one day Bart(Client) comes to Mr. X and said:

Bart: Hi Mr. X! I want an application for the DOK community.
Mr. X: Definitely Bart! What are the features that you would like to see in the application?
Bart: Yeah! The primary feature is it has to collect all the comments from all socials of dok community.
Mr.X: Fine! I will start working on your application.

After a year… He delivers the project in one go.

Mr.X: Hi Bart! Your application is ready.
Bart: Yeah well, Why don’t you add another feature like collect all comments and filter the most common comments.

This would be frustrating. Because developing software may take months or years.

But what if happened like this. While Mr.X is doing his application Bart may call or email him that I want this so and so feature.

So this is where agile comes into play:

Agile is a time-boxed, iterative approach to software delivery that builds software incrementally from the start of the project, instead of trying to deliver it all at once near the end.
Scrum is a framework of rules, roles, events, and artifacts used to implement Agile projects. It is an iterative approach, consisting of sprints that typically only last one to four weeks. This approach ensures that your team delivers a version of the product regularly.
A sprint in Scrum is a short period of time wherein a development team works to complete specific tasks, milestones, or deliverables. Sprints, also referred to as “iterations,” essentially break the project schedule into digestible blocks of time in which smaller goals can be accomplished.
It works by breaking projects down into little bits of user functionality called user stories, prioritizing them, and then continuously delivering them in short two-week cycles called iterations.
This means that data teams can publish new analytics in shorter increments, referred to as sprints, which significantly reduces wait times. Hence DataOps adopt an agile environment.

What are the challenges dataops trying to resolve?

The problem is that many organizations aren’t exactly clear on the most efficient approach to data collection and analytics. They often take a seemingly all-encompassing approach based on the principle of “we’ll collect the data and then figure out what to do with it,” that can do more harm than good.
They then have a data team who is supposed to miraculously turn garbage into gold, which generally requires far more effort than necessary and rarely leads to the desired results. Of course, this makes it virtually impossible to deliver actionable insights on a schedule that can keep up with the demands of a DevOps team that’s pushing to get their code to market.
DataOps takes this jumbled mess and turns it into a smooth process where data teams aren’t spending their time trying to fix problems. They aren’t wasting their time trying to turn poor raw data into something usable. Instead, they can focus on what matters, namely providing actionable insights.
DataOps ensures that the raw data coming in is useable, it ensures that the results are accurate, it focuses on the value of people and working together, and it keeps the data team at the center of the company’s strategic objectives. After all, they no longer take months to come up with the required insights.

Now here comes the big question of how Kubernetes help DataOps. Before that let us know what is kubernetes in a nutshell.

Let’s say you created an application(e.g Pokemon go) and decided to share it with the world. How can you do that? by deploying it to the internet. So you deployed this application into 5 different servers using docker. And your application starts getting massive traffic even you are not imagined. Millions of users started using. You are happy on one side simultaneously you are sad because you have to scale up the servers because 5 are not enough for such massive traffic. Now you need to scale up fast and make sure all containers restart if they die. And you will be gone out of control.
This is where Kubernetes comes into play.

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications.
Development teams and data science teams are increasingly interested in moving their build-test and model training environments to Kubernetes in order to drive efficiency and avoid vendor lock-in. When they make this migration, it is absolutely necessary for them to be able to take their clone-based processes with them so that they don’t lose all of the lifecycle efficiencies that they have achieved over the years. In fact, the lack of ability to clone a volume to a different namespace is often a showstopper. It is considered a best practice for different developers to work in different namespaces. This is done for security reasons, and there is typically no compromising when it comes to security. If clones are going to be included in the development lifecycle, the ability to clone workspaces to different namespaces is vital.

Every cluster has at least one worker node. Hence if a node fails, your application will still be accessible from the other nodes as in a cluster, multiple nodes are grouped.

Why should you use Kubernetes for DataOps?

DataOps has many things in common that relate them to Kubernetes. It serve the same business goals as devops when it comes to speeding up application software development, deployment, and delivery. In a nutshell, the DataOps practices focus on improving application production through software development workflow optimization powered by data-driven insights.

Kubernetes follows the same principles being a powerful tool that provides a rock-solid technological ground for enhancements on both software and human levels. DataOps engineers can leverage the accuracy of massive data assets for their BigData analytics through robust data orchestration, timely updates, persistent data storage, and streamlined data pipelines across platforms and departments.

K8s is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

The benefits Kubernetes brings to organizations in slightly more generic concepts. This is due to the fact that K8s can positively affect the workflows of various teams and departments at a time. What’s more, the benefits have an accumulative effect as you can capitalize on utilizing Kubernetes in the long run. Because of K8s open-source ecosystem.

 > Service deployment and adjustment.
 > Storage Alternatives.
 > Self-healing.
 > Automatic Selection.
 > Recovery Mechanisms.
 > Constant Monitoring. and many more...

Kubernetes environment is persistently becoming the leading platform for cloud computing. Consumers looking for dynamically-scheduled container-oriented DataOps solutions should consider Kubernetes-as-a-Service.

Just like Devops, DataOps is also a methdology it's not a tool.

Top DataOps Projects

Genie
Piper
Airflow
Naveego
Delphix Delphix is the industry leader in Data Ops and provides an intelligent data platform that accelerates digital transformation for leading companies around the world. Uses kubernetes.

Finally...

There is no doubt that Kubernetes is a true game-changer in today's software development. The platform helps you boost the overall performance and productivity when it comes to efficiently managing containerized services and workloads. Not to mention its huge time- and cost-saving potential. With Kubernetes at your fingertips, you can drastically enhance the efficacy of your DevOps, DataOps, and GitOps workflows while creating automated CI/CD pipelines.