python - how to find a string existing in file using mmap -
i have function take list of urls , save in file based on first letter of url. first, removed http:// , filter first letter , join first letter file extension , fine file search whether url exists in file if exists skip if not write in file , append diff_url_list here source code
def checkdiffurls(url_list): import mmap diff_url_list=[] file_extension="txt" urls in url_list: temp_url=urls.replace("http://","") url_head=temp_url[0][0] path=".".join((url_head,file_extension)) file=open("urls/"+path,'w') file_read = mmap.mmap(file.fileno(), 0, access=mmap.access_read) if urls not in file_read: file.write(urls) diff_url_list.append(urls) file.close() return diff_url_list
it gives me error mmap length greater file size can fix bug? thank you. appreciate that.
Comments
Post a Comment