Python 3 Script to Find Duplicate or Repeating Lines and Remove it in Text File Full Tutorial For Beginners

Python 3 Script to Find Duplicate or Repeating Lines and Remove it in Text File Full Tutorial For Beginners

 

Welcome folks today in this blog post we will be finding duplicate or repeating lines in text file and also we will be removing  it in python. All the full source code of the application is shown below

 

 

 

 

Get Started

 

 

 

In order to get started you need to make an app.py file and copy paste the following code

 

app.py

 

 

lines_seen = set() # holds lines already seen
outfile = open("output.txt", "w")
for line in open("file.txt", "r"):
    if line not in lines_seen: # not a duplicate
        outfile.write(line)
        lines_seen.add(line)
outfile.close()

 

 

Here in this snippet of python code you just need to replace the input and output text file paths respectively. And then if you execute this python script as shown below

 

python app.py

 

And now you can see our input text file which is file.txt which is shown below

See also  Python 3 Tkinter Bakery Management System (Login & Register) Using MySQL Database GUI Desktop App Full Project For Beginners

 

 

 

 

And now you see the output file which doesn’t contain the duplicate or repeated lines as shown below

 

 

 

Leave a Reply