Logsight SDK Python

README

https://img.shields.io/pypi/pyversions/logsight-sdk-py.svg

Logsight SDK Python is a set of open source libraries, tools, documentation, and code samples that provides an easy, lightweight solution to instrument the source code of custom applications so that you can monitor their behavior and performance with logsight.ai.

Logsight SDK uses a client/server architecture and communicates via HTTP with the AI-driven log analytics platform logsight.ai.

Main advantages:

Ease of use
No 3rd-party dependencies
Easily portability to other programming languages

Main functionalities:

Create and delete applications
Send log data records to logsight.ai
Retrieve incident reports
Analyze the quality of logs
Compare data logs

The Logsight SDK Python package is deployed to the following external platforms:

Test Python Package Index (TestPyPI): Test PyPI
Python Package Index (PyPI): PyPI
Documentation: docs

CHANGELOG

0.2.6 (2022-10-12)

Added env support for tags

0.2.5 (2022-06-18)

Replaced private emails
Added unittests for incidents
Removing application completely

0.2.2 (2022-06-09)

Improved testing

0.2.1 (2022-06-09)

SDK modified to support new Logsight API
Updated unittests to support new API

0.1.26 (2022-06-02)

File upload is no longer supported. Its being reworked and available once again in the next releases.
Sending logs no longer requires sending applicationID. Sending logs can be achieved with applicationName and without creating application.
Compare and Logger, are changed to fit these changes.

Quickstart

09/06/2022, 30 minutes to complete

Get started with the Logsight SDK for Python and client library to verify deployments. Follow these steps to install the package and start using the algorithms provided by logsight.ai service. The verification library enables you to evaluate the risk of a deployment of a new version of an application by automatically using deep learning models trained on millions lines of code, regardless of the underlying IT system, failure scenario, or data volume.

Use the Logsight SDK for Python to:

Send data logs to your logsight.ai account
Verify deployments using application data logs
Retrieve and display the verification risk of your new deployment

Prerequisites

logsight.ai subscription (create one for free)
You will need the login and password to paste into the code below, later

Setting up

Create a directory

Create a directory to store your exercise:

$ mkdir quick_guide
$ cd quick_guide

Create a virtual env

Create a Python virtual environment to decouple and isolate the packages we will install from you environment.

$ python3 -m venv venv
$ source venv/bin/activate

Prepare code file

You can start with an empty Python file:

$ touch quick_guide.py

Alternatively, you can download the Python file directly from git:

$ curl https://raw.githubusercontent.com/aiops/logsight-sdk-py/main/docs/source/quick_guide/quick_guide.py --output quick_guide.py

Install the client library

Install the Incident Detector client library for python with pip:

$ pip install logsight-sdk-py

or directly from the sources:

$ git clone https://github.com/aiops/logsight-sdk-py.git
$ cd logsight-sdk-py
$ python setup.py install

Create environment variables

Using the password from your subscription, create one environment variables for authentication:

LOGSIGHT_PASSWORD - Password for authenticating your requests
LOGSIGHT_EMAIL - Email associated with your subscription

Copy the following text to your bash file:

$ export LOGSIGHT_PASSWORD=<replace-with-your-password>
$ export LOGSIGHT_EMAIL=<replace-with-your-email>

After you add the environment variables, you may want to add them to ~/.bashrc.

For the impatient

mkdir quick_guide
cd quick_guide
python3 -m venv venv
source venv/bin/activate
curl https://raw.githubusercontent.com/aiops/logsight-sdk-py/main/docs/source/quick_guide/quick_guide.py --output quick_guide.py
pip install logsight-sdk-py
unset LOGSIGHT_PASSWORD LOGSIGHT_EMAIL
export LOGSIGHT_PASSWORD=mowfU5-fyfden-fefzib
export LOGSIGHT_EMAIL=logsight.testing.001@gmail.com
python quick_guide.py

Code example

The following code snippets show what can be achieved with the Logsight SDK client library for Python:

Authenticate the client
Set tags
Attach the logger
Log logging statements
Verify the new deployment
Show the results of the verification

Load packages

Load the various packages used in this guide.

import sys
import time
import logging

from logsight.config import set_host
from logsight.exceptions import InternalServerError
from logsight.authentication import LogsightAuthentication
from logsight.logger.logger import LogsightLogger
from logsight.compare import LogsightCompare

Authenticate the client

To enable client authentication, set your LOGSIGHT_PASSWORD and LOGSIGHT_EMAIL. If you use an on-prem deployment, setup the endpoint of your logsight system using function set_host.

EMAIL = os.getenv('LOGSIGHT_EMAIL') or 'logsight.testing.001@gmail.com'
PASSWORD = os.getenv('LOGSIGHT_PASSWORD') or 'mowfU5-fyfden-fefzib'
set_host("https://demo.logsight.ai/api/v1/")

auth = LogsightAuthentication(email=EMAIL, password=PASSWORD)

Attach the logger

Add logsight.ai logging handler to your logging system:

handler = LogsightLogger(auth.token)
handler.setLevel(logging.DEBUG)

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

Set tags

You can set tags using a dictionary

tags = {'service': 'redis', 'version': 'v1'}

or by using environment variables. To set tags as environment variables you need to use the prefix LOGSIGHT_TAG.

$ export LOGSIGHT_TAG_SERVICE=redis

from config import get_tags_from_env

env_tags = get_tags_from_env()  # {"service": "redis"}
handler.set_tags(tags=env_tags)

Execute Redis Version v1.1.1

We assume you are a core developer of Redis in-memory data structure store.

Run v1.1.1 of your Redis application
Logs are tagged with: service=redis and version=v1.1.1
Logs generated are transparently sent to logsight.ai

print('Redis running (v1.1.1)')
tags_1 = {'service': 'redis', 'version': 'v1.1.1'}
handler.set_tags(tags=tags_1)
for i in range(10):
    logger.info(f'Connecting to database (instance ID: {i % 4})')
    logger.info(f'Reading {i * 100} KBytes')
    logger.info(f'Closing connection (instance ID: {i % 4})')
handler.flush()

Your Redis deployment runs for several months without any problems. It is deemed reliable.

Execute Redis version v2.1.1

You implement a few new features for Redis. Your new version is v2.1.1.

Now, you run v2.1.1 of your Redis application in pre-production
Logs are tagged with: service=redis and version=v2.1.1
Logs generated are transparently sent to logsight.ai

print('Redis running (v2.1.1)')
tags_2 = {'service': 'redis', 'version': 'v2.1.1'}
handler.set_tags(tags=tags_2)
for i in range(15):
    logger.info(f'Connecting to database (instance ID: {i % 4})')
    logger.info(f'Unable to read {i * 100} KBytes')
    logger.error(f'Underlying storage is corrupted')
    logger.info(f'Closing connection (instance ID: {i % 4})')
handler.flush()

Verify New Release of Redis

You call the Compare function of Logsight to verify the new deployment

print('Calculate new deployment risk')
comp = LogsightCompare(auth.token)
result = {}
retry = 5
while retry:
    try:
        result = comp.compare(baseline_tags=tags_1, candidate_tags=tags_2)
        break
    except InternalServerError as e:
        print(f'Trying in 5s (#{retry})')
        time.sleep(5)
        retry -= 1

Show verification results

Display the deployment risk and access the webpage with the verification results

print(f'Deployment risk: {result["risk"]}')
print(f'Report webpage: {result["link"]}')

You can copy the url to your browser to see the results of the evaluation.

Run the application

Run the Python code from your quick_guide directory.

$ python quick_guide.py

Applications

Authentication

class logsight.authentication.LogsightAuthentication(email, password)

Bases: APIClient

__init__(email, password)

Class to authenticate users.

Parameters

email (str) – Email associated with the subscription.
password (str) – Password key associated with the subscription.

property token

Gets a token.

Returns: Access token
Return type: token (str)

property user_id

Gets the user id.

Returns: Identifier of the user
Return type: user_id (str)

Compare

class logsight.compare.LogsightCompare(token)

Bases: APIClient

__init__(token)

Class to compare logs.

Parameters: token (str) – Access token.

compare(baseline_tags, candidate_tags, log_receipt_id=None)

Compares the logs on an application.

Parameters

baseline_tags (dict) – Tags of the baseline logs.
candidate_tags (dict) – Tags of the candidate logs.

Returns

dict.

{

“addedStatesFaultPercentage”: 0, “addedStatesReportPercentage”: 0, “addedStatesTotalCount”: 0, “baselineLogCount”: 0, “baselineTags”: {

”additionalProp1”: “string”, “additionalProp2”: “string”, “additionalProp3”: “string”

}, “candidateChangePercentage”: 0, “candidateLogCount”: 0, “candidateTags”: {

”additionalProp1”: “string”, “additionalProp2”: “string”, “additionalProp3”: “string”

}, “compareId”: “string”, “deletedStatesFaultPercentage”: 0, “deletedStatesReportPercentage”: 0, “deletedStatesTotalCount”: 0, “frequencyChangeFaultPercentage”: {}, “frequencyChangeReportPercentage”: {}, “frequencyChangeTotalCount”: 0, “link”: “string”, “recurringStatesFaultPercentage”: 0, “recurringStatesReportPercentage”: 0, “recurringStatesTotalCount”: 0, “risk”: 0, “totalLogCount”: 0

}

Raises

BadRequest – if the app_name is invalid, it is duplicated, or too the maximum number of applications has been reached
Unauthorized – If the private_key is invalid.

get_comparison_id(comp_id)

Get comparison with id comp_id.

Parameters

comp_id (string) – Comparison Id.

Returns

dict.

{

“listCompare”: [

{
“_id”: “string”, “_source”: {}

}

]

}

ls_comparisons()

List comparisons stored.

Returns

dict.

{

“listCompare”: [

{

“_id”: “string”, “_source”: {

”baseline_tags”: {
“additionalProp1”: “string”, “additionalProp2”: “string”, “additionalProp3”: “string”

}, “candidate_tags”: {

”additionalProp1”: “string”, “additionalProp2”: “string”, “additionalProp3”: “string”

}, “risk”: 0, “severity”: 0, “status”: 0, “timestamp”: “string”

}

]

}

rm_comparison_id(comp_id)

Remove comparison with id comp_id.

Parameters

comp_id (string) – Comparison Id.

Returns

dict.

{: “compareId”: “string”

}

set_status(comp_id, status)

Set the status of a comparison with id comp_id.

Parameters

comp_id (string) – Comparison Id.
status (int) – Comparison status.

Returns

dict.

{: “compareId”: “string”

}

Logger

Logs

Users

class logsight.users.LogsightUsers

Bases: APIClient

__init__(): Class to manage users.

create(email, password)

Creates a new user.

Parameters

email (str) – Email associated with the subscription.
password (str) – Password key associated with the subscription.

Returns

Identifier for the user created

Return type

userId (str)

delete(user_id, token)

Deletes a new user.

Parameters: userId (str) – Identifier for the user

Exceptions

exception logsight.exceptions.APIException(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: Exception

__init__(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict) → None

Problem exception as defined in RFC 7807 (https://tools.ietf.org/html/rfc7807).

Parameters

status – HTTP status code generated by the remote server.
title – Short, human-readable title for the general error type; the title should not change for given types.
detail – Human-readable description of the specific error.
type – URL to a document describing the error condition (optional, and “about:blank” is assumed if none is provided; should resolve to a human-readable document).
instance – This optional key may be present, with a unique URI for the specific error; this will often point to an error log for that specific response.
**kwargs – additional context information

exception logsight.exceptions.BadGateway(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'the server, while acting as a gateway or proxy,\n received an invalid response from the upstream server.'

status_code = 502

exception logsight.exceptions.BadRequest(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'BadRequest: the server cannot or will not\n process the request due to something that is perceived\n to be a client error.'

status_code = 400

exception logsight.exceptions.Conflict(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'Conflict: a request conflict with current state\n of the target resource.'

status_code = 409

exception logsight.exceptions.DataCorruption(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'DataCorruption: the client was unable to parse a\n data structured received from the server.'

exception logsight.exceptions.Forbidden(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'Forbidden: the server understands the request\n but refuses to authorize it.'

status_code = 403

exception logsight.exceptions.InternalServerError(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'InternalServerError: the server has encountered\n a situation it does not know how to hand.'

status_code = 500

exception logsight.exceptions.LogsightException(message=None, **kwargs)

Bases: Exception

Base Logsight Exception

__init__(message=None, **kwargs)

message = 'An unknown exception occurred.'

exception logsight.exceptions.NotFound(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = "NotFound: the server can't find the requested\n resource."

status_code = 404

exception logsight.exceptions.ServiceUnavailable(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'ServiceUnavailable: the server is not ready to\n handle the request.'

status_code = 503

exception logsight.exceptions.Unauthorized(type_: Optional[str] = None, title: Optional[str] = None, status: Optional[int] = None, detail: Optional[str] = None, message: Optional[str] = None, instance: Optional[str] = None, **kwargs: Dict)

Bases: APIException

message = 'Unauthorized: the request has not been applied\n because it lacks valid authentication credentials for the\n target resource.'

status_code = 401

logsight.exceptions.from_dict(data: Dict[str, Any]) → APIException

Create a new APIException instance from a dictionary.

This uses the dictionary as keyword arguments for the APIException constructor. If the given dictionary does not contain any fields matching those defined in the RFC7807 spec, it will use defaults where appropriate (e.g. status code 500) and use the dictionary members as supplemental context in the response.

Parameters: data – The dictionary to convert into an APIException exception.
Returns: A new APIException instance populated from the dictionary fields.

Logsight SDK Python

README

CHANGELOG

0.2.6 (2022-10-12)

0.2.5 (2022-06-18)

0.2.2 (2022-06-09)

0.2.1 (2022-06-09)

0.1.26 (2022-06-02)

Quickstart

Prerequisites

Setting up

Create a directory

Create a virtual env

Prepare code file

Install the client library

Create environment variables

For the impatient

Code example

Load packages

Authenticate the client

Attach the logger

Set tags

Execute Redis Version v1.1.1

Execute Redis version v2.1.1

Verify New Release of Redis

Show verification results

Run the application

Applications

Authentication

Compare

Logger

Logs

Users

Exceptions

Indices and tables