Davinci vs. Python (PyCon Conference, March 10, 2017, Bratislava)

In March, Bratislava held (for the second time already) a nice conference dedicated to Python programming language - PyCon (link). You may ask yourself why would a member of the Cloud team go to a developers' conference? Well, I would answer that Python is not just a programming language, it’s more like a Swiss knife. Let me explain to you why I think so, and after you read my blog, you can decide whether I’m right or wrong. 

  

PyCon 2017

Every year, PyCon brings a broad selection of topics, which are not always directly connected to the development of enterprise applications (as somebody may think). You can learn a lot about new trends in IT and the current state of technology innovations in Slovakia. Also, it’s a conference organized by the Python community and it is filled with friendly people. PyCon provided many topics also this year, and I picked a few of them that interested me the most. Let me introduce some of them to you. 

 
What I really liked about the conference this year was the auditorium dedicated to education. The aim of it was to introduce Python to teachers, for example, as an alternative course of study to Pascal. I believe that everybody who studied IT encountered with the Pascal language at least once in their lifetime (yes, that boring blue DOS IDE). Isn’t it the time already to get rid of these legacy teaching procedures? Honestly, I believe that you have to be really dedicated to your IT future not to be discouraged by this development tool in the times of fancy user interface. Python would be a nice alternative to Pascal. Even my university teacher used to say that you can even write poetry in Python :).

Another (a little related) topic discussed at the conference was the Internet of Things (IoT). Personally, I'm an IoT hobbyist. But isn’t it nice to do some simple wiring, a little coding, and then finally shout “it’s alive!”? There was a workshop and a lecture at PyCon 2017 about a small open source embedded board NodeMCU (of size of a standard USB stick) for a price less than 3.50€! And guess what, you can code it in Python. Well, microPython to be precise. It has an embedded Wi-Fi chip on-board. Now imagine all the possibilities! For example, I work on a project for an intelligent garden, and I want to water the plants with a click of a button on my smartphone. Just a simple board near the watering system communicating via a Wi-Fi and a web browser with a user interface would be enough. People shouldn't be afraid of using the IoT and enjoy all the benefits it can bring. Having these kinds of projects at the University, I would be super happy and even more interested in IT. 

 

Also, there has been a lot said about web services at PyCon 2017. These days, web services are a big thing. Python has a few really nice frameworks for web development. You can use simple Flask to create REST API, or more complicated and enterprise oriented Django that is used by many big names worldwide. This gives you an opportunity to build simple web page with API in a few steps. Now try to imagine a connection with embedded devices. Again, unlimited options. 

I really liked a certain presentation by Peter Hanečák from Slovensko.Digital (and some others) on Open data. What is that? I didn’t know that before either. Apparently, it’s quite a term now. Did you know that Slovakia has an API that can provide you with all invoices paid by the government? And of course many more. Idea is that why would the state spend millions to get data from various fields, and not provide them via an API to others? This means that you are able to code an application that will help you to manage bureaucracy from the comfort of your home. Sounds good to me.  

The conclusion

So, where to use Python according to PyCon? Web development, games development, science (even space science!), IoT, automation, machine learning, OS development, education, AI, testing, encryption, hacking, networking and many more. As a proper fan of Python, I found ways to utilize it at my job to make my work easier. Now, I will introduce some Python usage examples of our own.  

 

Python in practice: where do we use it at work? 

  

APIs 

Most of the time at work, we try to make two different applications talk to each other. Basically every software has an API that may be used for this purpose. We used Python to create a very simple application that makes two different APIs talk to each other even if they are not friends or they are after a very nasty argument :).

Incidents handling 


One of the most important Python application we use is for incidents handling. Well, you can't avoid incidents in the enterprise environment, and you should be able to handle them somehow. For this purpose we use the PagerDuty application, which receives alerts from various monitoring applications to keep us notified when something is broken. But what about measuring SLAs, communicating with the customer or creating reports? That’s where we use Jira. We keep incidents as Jira tickets, which can be configured with various parameters (also the customer can see or create incidents related to his project). So where is the problem? Imagine that you have PagerDuty installed on your smartphone. New incident comes and you have to log into your computer, create a Jira issue and fill in all the parameters. If you are near your computer then it doesn’t seem like a problem. But if you are away, and 8 new incidents come, and you have 10 minutes to report them, so your customer can see that his precious application is in good hands, then it’s starting to be a bit painful. 

That’s why we’ve created a Python application in between PagerDuty and Jira. Now, when a new incident comes, PagerDuty hits the Python API, Python API examines the root system that has reported the incident, parses payload, identifies which customer and which environment is affected, who is responsible for this project and so on, and creates a Jira issue that is fully filled with data. Now, the standby person acknowledges the incident, and the status of the Jira issue is automatically changed to “In progress” or “Incident resolved”. With a simple click of a button on a mobile application the Jira issue is resolved and we don't waste the SLA minutes. With a variety of configuration options from Jira, we have a strong tool to make a customer specific support desk, and at same time handle incidents much faster. 

Amazon AWS Lambda 

Amazon is lately aiming to serverless architecture, and is adopting its services for this purpose. The basic idea is pretty nice. Why maintain instances, load balancers and etc. when you can deploy your code into AWS Lambda? Especially when you have a simple application. Lambda as an AWS service came to our SysOPS lives just recently, but we saw its big potential instantly and we are already using it on the production environments. 

For example, developers designed one back-office application that should run batch jobs every 10 minutes and one at the end of the day. These two jobs run with different parameters, and also require different scaling options. Requirement was that they should not interrupt each other. This seemed like a perfect opportunity to try AWS Lambda. So how is it configured? 

AWS CloudWatch offers a possibility to create events. Imagine that as cron jobs in Linux. So that would be a solution for recurring batch jobs at a specific time. Events trigger the AWS Lambda function (of course running Python), which scales the application through AWS Autoscaling to a desired capacity and runs the back-office job with the correct parameters. Python application also takes care of the locking mechanism (simple lock file stored on AWS S3) so jobs won’t interrupt each other. For encryption we use AWS KMS.  

New Relic cooperation 

Another place where we use AWS Lambda is monitoring of applications via New Relic. New Relic offers a possibility to group applications or servers with specific alarm thresholds for CPU utilization, RAM usage and so on. But what if every application needs a specific group? Applications are assigned to a default group, and then you have to move them in a web application to a desired group. Also, when an instance is terminated, you have to deal with a fleet of zombie servers in New Relic. For automation purposes, we’ve created AWS Lambda running Python that solves this for us. It assigns an instance to its group when provisioning and when it’s terminated, it removes it from New Relic. This is how we do it. 

 

When an instance is scaled, AWS Autoscaling sends an AWS SNS notification with information about the instance. This message is then stored in AWS SQS queue, which triggers AWS Lambda running Python. Python application parses this message, and decides to which group it belongs. Then via a New Relic API it checks if the application or the instance is in the right group. If not, then it does all the dirty work to move it to the right place. The same procedure applies when an instance is terminated. AWS Lambda finds out to which groups the instance or the application is assigned, and moves it to the group with terminated instances. This group is cleaned twice a day by yet another Python script. 

Others 

This blog describes in more detail only the biggest projects. But I have to mention also a few other smaller applications for which we use Python: 

  • Generating Wiki pages on Jira with up-to-date IP addresses of instances running on specific projects. IPs are received directly from AWS every morning. 
  • Container automatization for Windows. 
  • Migration of logs from Kibana to Splunk. 
  • Checking of time to expiration of SLA timer on incidents with mail notifications to a standby person and the Cloud team. 
  • Generating SLA reports for HTTPS endpoints on all projects. 

 

The end 

Well, connecting two or more already functioning applications may create a strong tool. If you connect two sets of really awesome features, nothing bad can come out of it. Just remember that you don’t have to reinvent the wheel, just make applications you like to talk to each other in a reasonable way, and benefit from it.  

 

The very end 

So, why would a Cloud guy go to a developers' conference? Maybe now it makes a bit more sense to you. Cloud specialists or SysOPS need skills in basic programming or at least scripting. It will help you make your life easier, and save time for a nice game of foosball with your colleagues :). You may say that this can be done in any other programming language. Well, you are right. I’m not saying that Python is the Holy Grail, but it’s really easy to learn, understand, read, has a nice community, and it has a native support in Linux, which we are using in the majority of our applications.  

P.S.: If you have any (technical) questions concerning my blog, please, don’t hesitate to contact me at shopko@davincisoftware.sk. Thx!

  

Publikované: 24. apr 2017 12:23
  • Samuel Hopko

    Cloud Specialist

    Samo pracuje v Davinci na pozícii Cloud Specialist. Na starosti má zabezpečovanie infraštruktúry pre hladký beh nami vyvíjaného softvéru v Cloud prostredí. O túto technológiu sa zaujímal už počas vysokoškolského štúdia, teoreticky aj prakticky. Vo voľnom čase si rád zahrá Bacha na gitare. 

We want you

Do you see yourself working with us? Check out our vacancies. Is your ideal vacancy not in the list? Please send an open application. We are interested in new talents, both young and experienced.

Join us

Podobné články