Or syntax when parsing urls with regex & python

Question

Struggling with some regex here. I'll be looping through several urls but I cannot get the regex to how to recognize revenue or cost and grab the them both. Essentially the output would look something like this:

import re

url = ['GET /ca.gif?rb=1631&ca=20564929&ra=%n&pid=&revenue=224.00&cost=',
       'GET /ca.gif?rb=1631&ca=20564929&ra=%n&pid=&revenue=224.00',
       'GET /ca.gif?rb=1631&ca=20564929&ra=%n&pid=&revenue=224.00&cost=13']
values = []
for i in urls:
    values.append(re.search(r'(?<=revenue=)(.*?)(?=&|;)',url).group(0))

print values

[[224.00, ''],
 '224.00',
 [224.00, 13]]

I think the last list is your expected output.

Avinash Raj
– Avinash Raj

2015-06-05 01:19:41 +00:00
Commented Jun 5, 2015 at 1:19 — Avinash Raj
– Avinash Raj, Commented Jun 5, 2015 at 1:19

Avinash Raj · Accepted Answer · 2015-06-05 01:14:02Z

1

You need to use re.findall since re.search returns only the first match.

>>> for i in url:
        values.append(re.findall(r'(?:\brevenue=|\bcost=)(.*?)(?:[&;]|$)', i))


>>> values
[['224.00', ''], ['224.00'], ['224.00', '13']]

answered Jun 5, 2015 at 1:14

Avinash Raj

175k32 gold badges247 silver badges289 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

nhahtdh · Accepted Answer · 2015-06-05 04:46:16Z

Use urlparse.urlparse to parse the URL, and urlparse.parse_qs to parse the query string.

from urlparse import urlparse, parse_qs

reqs = ['GET /ca.gif?rb=1631&ca=20564929&ra=%n&pid=&revenue=224.00&cost=',
        'GET /ca.gif?rb=1631&ca=20564929&ra=%n&pid=&revenue=224.00',
        'GET /ca.gif?rb=1631&ca=20564929&ra=%n&pid=&revenue=224.00&cost=13']

urls = [re.split(' +', s, 1)[1] for s in reqs]

kv = [parse_qs(urlparse(url).query) for url in urls]

values = [(e.get('revenue'), e.get('cost')) for e in kv]
# values = [{'revenue': e.get('revenue'), 'cost': e.get('cost')} for e in kv]

Sample output (parse_qs provides a list of values for every key, since the query may contain duplicate keys):

[(['224.00'], None), (['224.00'], None), (['224.00'], ['13'])]

The values line is not necessary. You can use the kv dict directly.

If you have to deal with invalid input, the list comprehension with urls and kv has to be rewritten as a loop:

For urls, you need to check and filter out entries without HTTP method
For kv, you need to add try catch for urlparse to catch invalid syntax.

Collectives™ on Stack Overflow

Or syntax when parsing urls with regex & python

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related