Python 3 Twitter Web Scraper to Get Tweets, Likes, Retweets, Following, Followers of User in Command Line

 

 

A simple and unlimited Twitter scraper with python.

Recently, Twitter has banned almost every Twitter scraper. This repository presents an alternative tool to scrape Twitter based on 3 functions:

  • scrape: Scrapes all the information regarding tweets between two given dates, for a given language and list of words or account name, in the form of a csv file containing retrieved data (more storage methods will be added).
  • get_user_information: Scrapes users information, incluing number of following and followers, location and description.
  • get_users_followers and get_users_following: Scrapes followers and following accounts for a given list of users.

It is also possible to download the images showed in tweets by passing the argument save_images = True. If you only want to scrape images, it is recommended to set the argument display_type = image to show only tweets that contain images.

Authentication is required for scraping followers/following. It is recommended to log in with a new account, otherwise the account could be banned if the list of followers is very long. To log in to your account, you need to enter your username SCWEET_USERNAME and password SCWEET_PASSWORD in the .env file. You can control the wait parameter in the get_users_followers and get_users_following functions according to you internet speed.

Requirements :

pip install -r requirements.txt

Note : You must have Chrome installed on your system.

Results :

Tweets :

The CSV file contains the following features (for each tweet) :

  • ‘UserScreenName’ :
  • ‘UserName’ : UserName
  • ‘Timestamp’ : timestamp of the tweet
  • ‘Text’ : tweet text
  • ‘Embedded_text’ : embedded text written above the tweet. This can be an image, a video or even another tweet if the tweet in question is a reply
  • ‘Emojis’ : emojis in the tweet
  • ‘Comments’ : number of comments
  • ‘Likes’ : number of likes
  • ‘Retweets’ : number of retweets
  • ‘Image link’ : link of the image in the tweet
  • ‘Tweet URL’ : tweet URL

Following / Followers :

The get_users_following and get_users_followers in user file give a list of following and followers for a given list of users.

Usage :

Library :

The library is now available. To install the library, run :

pip install Scweet==1.8

After the installation, you can import and use the functions as follows:

Scrape top tweets with the words ‘bitcoin’, ‘ethereum’ geolocated less than 200 km from Alicante (Spain) Lat=38.3452, Long=-0.481006 and without replies:
The process is slower as the interval is smaller (choose an interval that can divide the period of time between, start and max date)

Scrape top tweets of with the hashtag #bitcoin, in proximity and without replies:
The process is slower as the interval is smaller (choose an interval that can divide the period of time between, start and max date)

Get the main information of a given list of users:
These users follow me on Twitter

**This function will return a list that contains : **
[“no. of following”,”no. of followers”, “join date”, “date of birth”, “location”, “website”, “description”]

Get followers and following of a given list of users Enter your username and password in .env file. I recommend you do not use your main account.
Increase wait argument to avoid banning your account and maximize the crawling process if the internet is slow. I used 1 and it’s safe.

Set your .env file with SCWEET_EMAIL , SCWEET_USERNAME and SCWEET_PASSWORD variables and provide its path

Terminal :

To run the script :

python scweet.py --words "excellente//car" --to_account "tesla" --until 2020-01-05 --since 2020-01-01 --limit 10 --interval 1 --display_type Latest --lang="en" --headless True

 

 

https://github.com/Altimis/Scweet

 

 

Leave a Reply