New Relic: Respond faster, optimize better, and build more perfect software
Introduction
The rise of MLOps (Machine Learning Operations) platforms and tools has been a significant development in the field of machine learning and artificial intelligence over the past few years. MLOps tools and platforms are aimed at bridging the gap between the development of machine learning models and their deployment in production environments. It involves a set of practices, processes, and tools that facilitate the smooth integration of machine learning into real-world applications, while ensuring reliability, scalability, and maintainability.
What is New Relic
New Relic is a software analytics company that specializes in providing performance monitoring and observability solutions for software applications.
What problems does it solve?
Monitoring, troubleshooting, and optimizing software applications’ performance, reliability, and user experience are essential for ML/AI companies for several reasons:
- Improved model performance: By monitoring and optimizing the performance of ML applications, companies can ensure that their models are running efficiently and producing accurate results.
- Resource management: ML applications often require significant computational resources, such as CPU, GPU, memory, and storage. Efficient monitoring and optimization help in minimizing resource usage and associated costs, thus ensuring cost-effective deployment and scaling of ML solutions.
- Faster issue resolution: ML applications can be complex, with several interconnected components, data pipelines, and dependencies. Monitoring and troubleshooting allow ML companies to quickly identify and address issues, minimizing downtime and ensuring the reliability and stability of their applications.
- Continuous improvement: ML models and applications need to be updated regularly to keep up with the evolving data and requirements. Monitoring and optimization provide valuable insights that can be used to refine models, improve algorithms, and streamline workflows, leading to better performance and outcomes over time.
By offering these services, New Relic enables companies to identify and address issues early, ensuring that their applications are running smoothly and efficiently. It helps businesses monitor, troubleshoot, and optimize their software applications’ performance, reliability, and user experience.
Value provided to customers
By using New Relic, ML/AI companies can gain valuable insights into their applications and infrastructure, enabling them to optimize performance, address issues proactively, and ensure the delivery of reliable and efficient ML/AI solutions and hence taking care of the above problems.
How to leverage New Relic in production environment ?
Use Case: Movie Recommendation
New Relic can be highly useful in a production environment such as a movie streaming and recommendation platform like Netflix in the following ways-
Monitoring recommendation engine performance: In a movie recommendation system, the performance of the recommendation model is crucial for providing accurate and relevant suggestions.
Example: New Relic’s Application Performance Monitoring (APM) and error logs functionalities can be used to monitor the performance of the recommendation engine, helping identify issues like slow response times, high error rates, or increased latency or exceptions.
In the above example I can see two errors as response to GET request in a movie recommendation task, one arising due to ‘occupation’ not found and one due to the request timing out.
Infrastructure monitoring: A movie recommendation platform will also require a robust infrastructure to handle large amounts of data along with many user requests.
Example: Under the Infrastructure feature, you can help track resource utilization, network performance, and other key metrics like CPU% usage, Load Average, and % free memory, that helps ensure a high-quality user experience. If a sudden surge in user traffic is causing server overloads or network latency, Infrastructure Monitoring can help identify the affected components and enable the team to address the issue before it impacts user experience.
Monitoring user experience: Ensuring availability, and a smooth and responsive user experience is critical for retaining and engaging users on a movie streaming platform.
Example: New Relic’s Browser and Mobile Monitoring solutions can help monitor frontend performance, including page load times, user interactions, and potential issues that could affect user experience.If users are experiencing slow load times when browsing the movie catalog, Browser Monitoring can help identify the root cause, such as inefficient JavaScript code or heavy media files, allowing the team to optimize the frontend for faster performance.
Monitoring API performance: Movie streaming platforms often rely on APIs for data exchange between various services, such as content delivery, user authentication, and billing. New Relic’s APM can be used to monitor API performance, ensuring that these services are running efficiently and reliably.
Example: If the content delivery API is experiencing high error rates or slow response times, APM can help identify the issue, enabling the team to address it and maintain a seamless content streaming experience for users.
Log management and analysis: In a complex ML-powered system like a movie streaming platform, log data can provide valuable insights into the system’s health and performance. New Relic’s Log Management solution can help collect, analyze, and visualize log data from various sources, making it easier to identify and troubleshoot issues.
Example: If there’s an unexpected drop in the accuracy of movie recommendations, Log Management can help correlate log data from the recommendation engine, data pipelines, and infrastructure components to identify the root cause and guide the team in resolving the issue.
By using New Relic in a production ML system like a movie streaming and recommendation platforms, teams can gain comprehensive insights into the application and infrastructure performance, enabling them to proactively address issues and maintain a high-quality user experience.
Strengths of the tool:
- Comprehensive monitoring: New Relic provides end-to-end monitoring across the entire software stack, including applications, infrastructure, and user experience.
- Real-time insights: The platform delivers real-time performance monitoring and analytics, enabling teams to detect and troubleshoot issues as they occur, minimizing the impact on end-users.
- Broad language and framework support: New Relic supports a wide range of programming languages, frameworks, and platforms, making it compatible with diverse technology stacks and suitable for various types of applications and industries.
- Customizability: New Relic allows users to create custom metrics, dashboards, and alerts tailored to their specific needs, providing a personalized view of the system’s performance and helping teams focus on the most relevant indicators.
- Ease of integration: New Relic offers numerous integrations with popular cloud providers like AWS, GCP, Azure, and DevOps tools, and other services, making it easy to incorporate the platform into existing workflows and environments.
- Collaboration within cross functional teams: The platform provides a unified view of application and infrastructure performance, facilitating collaboration among cross-functional teams and enabling them to work together effectively in addressing issues and improving system performance.
Limitations:
- Cost: New Relic can be expensive, particularly for small businesses and startups with limited budgets. The pricing model can become costly as the number of monitored services, users, and data volume increases.
- Complexity: The platform offers a wide range of features and capabilities, which can make it challenging for some users to navigate and use effectively. It may require a learning curve for users who are new to application performance monitoring and observability.
- Limited support for certain technologies: While New Relic supports many languages, frameworks, and platforms, it may not provide full support for some specific technologies or use cases. Users working with less common technologies may face limitations in monitoring and observability.
- Overhead and performance impact: Although New Relic aims to minimize the performance overhead associated with monitoring, some users have reported performance impacts when using the platform. This can be a concern, especially in resource-constrained environments or when monitoring highly sensitive applications.
How to integrate New Relic in your current ML project?
- Create a New Relic account: Go to the New Relic sign-up page (https://newrelic.com/signup) and create an account using your email address or a supported single sign-on (SSO) provider.
- Log in to your New Relic account and access the New Relic One dashboard.
- Install the New Relic Python agent:
- In your Linux VM, make sure you have Python and pip (the Python package manager) installed.
- Activate your virtual environment using
source venv/bin/activate- Install the New Relic Python agent using pip
pip install newrelic- This installs the agent and the required dependencies.
4. Generate the New Relic configuration file:
- To generate the New Relic configuration file, run the following command, replacing YOUR_LICENSE_KEY with your actual New Relic license key, and YOUR_APP_NAME with the desired name for your movie recommendation application:
newrelic-admin generate-config YOUR_LICENSE_KEY newrelic.ini- Edit the newrelic.ini file if you want to customize any agent settings, such as logging or data collection.
- Copy your newrelic.ini file to your VM
After installation, you’ll see the following in your terminal-
3. Integrate New Relic with your Flask application:
- Open your Flask application’s main Python file (e.g., app.py).
- Import the newrelic.agent package at the beginning of the file:
- Before the Flask application instance is created, initialize the New Relic agent by adding the following line, specifying the path to your newrelic.ini configuration file:
- newrelic.agent.initialize(‘newrelic.ini’)
- To monitor your Flask application, wrap your app instance with the newrelic.agent.wsgi_application() decorator:
newrelic.agent.initialize('newrelic.ini')
app = newrelic.agent.wsgi_application()(app)
3. Integrate logs:
- Use this command into your host to enable infrastructure and logs metrics.
curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash && sudo NEW_RELIC_API_KEY=NRAK-OFHUHP4SF5GF3F52KZ6UBKM5SR7 NEW_RELIC_ACCOUNT_ID=3882989 /usr/local/bin/newrelic install -n logs-integration4. Start your Flask application:
- Start your Flask application with the New Relic agent by running the following command:
NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program python app.py- Replace app.py with the name of your Flask application’s main Python file, if necessary.
- This command starts your Flask application with the New Relic Python agent, which will begin monitoring your application’s performance and sending data to your New Relic account.
NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program flask --app main run --host=0.0.0.0 --port=80825. Monitor your movie recommendation application in New Relic:
- Log in to your New Relic account and access the New Relic One dashboard.
- In the “Entity explorer” tab, you should see your movie recommendation application listed under the name you specified in step 3.
- Click on the application name to view performance metrics, such as response times, throughput, and error rates.
Now, New Relic is set up for your movie recommendation project, and you can monitor, troubleshoot, and optimize your Flask web application using the New Relic dashboard.
You can also set up your query dashboard, and query on the real time data. In this example, it is counting all the infrastructure events.
Hope you enjoyed reading this!
References:
