TomsBlog

Deluge RPC Client Python

Thu, 21 Feb 2019 23:57:14 +0000

Installing

Deluge is a torrent client written in python. It however has an RPC API so you can interact with it remotely. Using this API you can add, delete and gather status reports on torrents, you can therefore produce a nice looking frontend for it. It is worth pointing out that there is a web interface for it already called deluge-web.

The package on Ubuntu is simply deluge:

sudo apt update
sudo apt install -y deluge

There is a python client to make it easier to interact with the deluge RPC client which we will use.

This can be installed through pip:

pip3 install deluge-client

Basic Code

The following is the minimum code you need to connect to the RPC server:

from deluge_client import DelugeRPCClient

delugeClient = DelugeRPCClient('127.0.0.1', 58846, 'username', 'password')
delugeClient.connect()
if not delugeClient.connected:
    raise Exception("Error connecting to Deluge Daemon")

Running a function

The following doc shows the RPC call: https://deluge.readthedocs.io/en/develop/reference/rpc.html

The following snippet from that doc shows the RPC function call format:

[[request_id, method, [args], {kwargs}], …]

request_id (int)

An integer determined by the client that is used in replies from the server. This is used to ensure the client knows which request the data is in response to. Another alternative would be to respond in the same order the requests come in, but this could cause lag if an earlier request takes longer to process.

method (str)

The name of the remote method to call. This name can be in dotted format to call other objects or plugins methods.

args (list)

The arguments to call the method with.

kwargs (dict)

The keyword arguments to call the method with.

The request_id is taken care of by the plugin therefore all we have to put in is the method the args and the kwargs. A list of methods and their inputs can be found here:

https://deluge.readthedocs.io/en/develop/reference/api.html

We can now get an object of torrents currently in Deluge.

torrent = delugeClient.call('core.get_torrents_status', {}, [])

This returns a python object of each torrent, for each torrent there are a lot of keys but here are some of the more interesting ones:

name
state
progress
total_size
download_payload_rate
upload_payload_rate
eta

It is worth noting that the keys in the object that is returned are in binary and not string format. You therefore need to convert them by looping through and converting bytes to str. The below code is an example of this

for key1, value1 in torrent.items():
	torrents=[]
	new_torrent={}
	for key, value in value1.items():
		if isinstance(value, bytes):
			value = str(value).replace("b'", "").replace("'", "").replace('b"', '').replace('"', '')
		new_torrent[str(key).replace("b'", "").replace("'", "")] = value
		torrents.append(new_torrent)

PCAP Python analyser with JSON d3 visualisation

Sat, 09 Dec 2017 11:14:51 +0000

I have been in a few environments where installing a full blown PCAP analyser is just not possible. Therefore I created a Python script that will go through a PCAP file, split up IP flows and spit it out in a JSON format. This depicts TCP and UDP flows with the amount of traffic uploaded and downloaded, it also attempts to map IP addresses to domains using DNS requests found in the same PCAP file.

Having it in the this format then allows it to be easily visualised using something like d3.

The code can be found here:

https://github.com/thomasweaver/pcap-visualizer

Using the HTML files you can visualise the traffic with the forced graph, colour cordinating the traffic and showing flows that have downloaded or uploaded more in a bolder line. The treemap allows you to drill down to IP address and port showing you the big hitters.

Requirements

Python Scapy is required which can be installed with pip:

pip install scapy

Other packages that are required but are installed by default with python:

os
sys
json
hashlib
copy

The HTML files get the d3 javascript files from the below URL’s: * d3js.org/d3.v4.min.js * d3js.org/d3-selection-multi.v1.js * d3js.org/d3.v3.min.js

Using

Locate the folder that contains your PCAP file and run the python script

pcap-analyzer.py C:\my-pcap-files\

The script will then loop through all the pcap files in this folder. Try to make sure the PCAP file are named appropriately so that they are parsed chronologically this will ensure the DNS mapping will be as accurate as possible.

This will create 2 JSON files: * output.json - This is the main json file which can be used to visualise the data * dnsMapping.json - This is the IP address to DNS name mappings. The script finds DNS requests in the pcap and maps IP addresses to domain names. This is the output of that mapping

You can then use the below command to launch a simple web server:

python -m SimpleHTTPServer

Then access the force map on the below url: http://127.0.0.1:8000/forced.html

and the treemap on: http://127.0.0.1:8000/treemap.html

The d3 Treemap visualisation is based on http://bl.ocks.org/ganeshv/6a8e9ada3ab7f2d88022 THe d3 forced graph is based on various examples on http://bl.ocks.org

Screenshots:

AWS send email with Lambda and SES

Thu, 10 Nov 2016 20:20:18 +0000

Lambda

Lambda is an AWS feature that allows you to run functions based on triggers such as API calls, SNS events or s3 file uploads. The advantage of Lambda is Amazon takes care of all the infrastructure for you so all you have to care about is the function. Amazon will elastically spin the infrastructure up and down to accommodate the load and even better will only charge you for the time your function is running. Nodejs, Python and Java are all supported, the below example is written in python but there are plenty of other language examples out there.

SES

SES is Amazons email offering allowing you send or receive email in your AWS account. It can be used for marketing campaigns or adhoc emails and basically any email needs. As with Lambda you only pay for what you use. To send emails from a domain you have to verify your domain first by adding a TXT record on your DNS, this is all explained when setting up SES in your console.

Example

In this example we set up a lambda function to send an email to a predefined address using SES. We have already set up SES and verified our domain by adding the TXT record provided.

The example code below does minimum user input validation and is meant only as an example.

import boto3
        
def lamda_sendMessage(event, context):
    if not (('subject' in event) and ('message' in event) and ('from' in event) and ('name' in event)):
        return {"code": 1, "message": "Must preovide all values"}
    if event['subject'] != "" and event['message'] != "" and event['from'] != "" and event['name'] != "":
        toEmail = "email@domain.com"
        fromEmail = "email@domain.com"
        replyTo = event['from']
        name = event['name']
        subject = event['subject'] + " - " + name
        message = event['message']

        client = boto3.client('ses')
        response = client.send_email(
			Source=fromEmail,
			Destination={
				'ToAddresses': [
					toEmail,
				],
			},
			Message={
				'Subject': {
					'Data': subject,
					'Charset': 'utf8'
				},
				'Body': {
					'Text': {
						'Data': message,
						'Charset': 'utf8'
					}
				}
			},
			ReplyToAddresses=[
				replyTo
			]
		)
			
        print response['MessageId']
        return {'code': 0, 'message': 'success'}

The above code uses the boto client to send an email using the SES “send_email” function. More on the SES boto API can be found here:

http://boto.cloudhackers.com/en/latest/ref/ses.html

Now the function is setup we need to give the role the function runs under access to SES. To do this go to “Identity and Access Management” then Roles and click on the Role associated with the function. Click the Policy and put the below JSON into the Policy replacing the ID with your ARN ID.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:eu-west-1:ID:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:eu-west-1:ID:log-group:/aws/lambda/function:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": "*"
        }
    ]
}

Now we have a Lambda function we need to trigger it, in this example we will be triggering it through an API. To do this we need to implement Amazons API Gateway.

API Gateway

Create an API and then add a resource, in our case we will call the Resource “sendmessage”. Add a POST type method, select lambda function then set your Lambda region and then your lambda function

This gives you the basic setup now you can deploy your API by clicking Actions->Deploy API. It will ask you which stage you want to deploy it to, just create a new one and deploy it. Go to Stages click on your stage you created previously and there will be a URL you can invoke the API with, it will look something like this:

https://{apikey}.execute-api.{location}.amazonaws.com/{stage}

This API can be called using a curl command similar to one below:

curl -i -XPOST 'https://{apikey}.execute-api.{location}.amazonaws.com/prod/sendmessage' -d '{"message": "sent from curl", "subject": "Sent from curl", "from": "email@domain.com", "name": "Test"}'

This should return some JSON with code 0 and a message of “success”.

CORS Consideration

Another consideration that needs to be made is how to call your API. In most cases you will want to call the API from some Javascript on your website.

This will not work by default due to “Access-Control-Allow-Origin” headers not being returned on the API call.

This can be enabled in the API gateway by going to Actions->Enable CORS by default this allows all Origins to query the API, you will want to change this to your own website.

Don’t forget once you have made a change to the API gateway you need to re-deploy the API for the changes to go live.

This example is a basic setup there are other considerations that need to be made such as API keys, rate limiting, input validation and response mappings.