# Web scraper BeautifulSoup

**URL:** <https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155>\
**Category:** Ignition\
**Created:** [May 2, 2023, 5:31pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155 "2023-05-02T17:31:27Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 2, 2023, 5:31pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/1 "2023-05-02T17:31:27Z")

</div>

Hello, I'm wanting to add beautifulsoup 3.2.2 on ignition 7.9 but ignition wont add it when i do file\>import any help would be great, thanks  
[BeautifulSoup3.2.2.zip](https://forum.inductiveautomation.com/uploads/short-url/4qjfSbxXgZH2NJef3TWi4EavnV1.zip) (22.0 KB)

---

<div class="post-metadata">

**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\
**Post date:** [May 2, 2023, 5:33pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/2 "2023-05-02T17:33:56Z")

</div>

[https://docs.inductiveautomation.com/display/DOC79/Libraries#Libraries-Importing3rdPartyLibraries](https://docs.inductiveautomation.com/display/DOC79/Libraries#Libraries-Importing3rdPartyLibraries)

---

<div class="post-metadata">

**Author:** ![JordanCClark](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/jordancclark/32/83362_2.png) [@JordanCClark](https://forum.inductiveautomation.com/u/JordanCClark)\
**Post date:** [May 2, 2023, 5:46pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/3 "2023-05-02T17:46:54Z")

</div>

If it's the one from this post (and looking at the file, I believe it is), then this was put together for 8.x

> [@Does anyone know of a Beautiful Soup J-python equivalent?](https://forum.inductiveautomation.com/t/does-anyone-know-of-a-beautiful-soup-j-python-equivalent/35719/4):
>
> Import to project library: [BeautifulSoup3.2.2.zip](https://forum.inductiveautomation.com/uploads/short-url/4qjfSbxXgZH2NJef3TWi4EavnV1.zip) (22.0 KB) Test using one of our power monitors: from BeautifulSoup import BeautifulSoup url = "http://192.168.140.31/15" soup = BeautifulSoup(system.net.httpGet(url)) t = soup.find('table') tr = t.findAll('tr') headers = ['meter', 'value'] data = [] for row in tr: td = row.findAll('td') if len(td) \> 0: data.append([col.text for col in td]) dataOut = system.dataset.toDataSet(headers, data) Resultant dataset: row | meter …

You can try unzipping it, then copying the contents of code.py into a blank script.

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 2, 2023, 7:38pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/4 "2023-05-02T19:38:13Z")

</div>

Open it in notepad and copy everything into a script? And where would I do the parsing, in the same script?

---

<div class="post-metadata">

**Author:** ![JordanCClark](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/jordancclark/32/83362_2.png) [@JordanCClark](https://forum.inductiveautomation.com/u/JordanCClark)\
**Post date:** [May 2, 2023, 7:39pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/5 "2023-05-02T19:39:09Z")

</div>

Yes to both. 🙂

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 2, 2023, 7:43pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/6 "2023-05-02T19:43:58Z")

</div>

i did then i opened script console and put this in  
from BeautifulSoup import BeautifulSoup

url = "[http://192.168.140.31/15](http://192.168.140.31/15)"  
soup = BeautifulSoup(system.net.httpGet(url))

t = soup.find('table')  
tr = t.findAll('tr')

headers = ['meter', 'value']  
data =   
for row in tr:  
td = row.findAll('td')  
if len(td) \> 0:  
data.append([col.text for col in td])

dataOut = system.dataset.toDataSet(headers, data)

but got the following error

"eback (most recent call last):  
File "", line 1, in   
ImportError: No module named BeautifulSoup"

---

<div class="post-metadata">

**Author:** ![Transistor](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/transistor/32/3380_2.png) [@Transistor](https://forum.inductiveautomation.com/u/Transistor)\
**Post date:** [May 2, 2023, 7:48pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/7 "2023-05-02T19:48:27Z")

</div>

Please edit your post, select the code block, and press the \</\> button to format it as code. This will preserve indentation and apply syntax highlighting.

---

<div class="post-metadata">

**Author:** ![JordanCClark](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/jordancclark/32/83362_2.png) [@JordanCClark](https://forum.inductiveautomation.com/u/JordanCClark)\
**Post date:** [May 2, 2023, 7:56pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/8 "2023-05-02T19:56:08Z")

</div>

- Show us where you put it. This will affect where your import happens.
- You're not going to get anything from my power meter. It doesn't go to the internet. 😉

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 2, 2023, 8:08pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/9 "2023-05-02T20:08:58Z")

</div>

oh sorry, here's where i put the code.py in

 ![Screenshot 2023-05-02 130757](https://us1.discourse-cdn.com/inductiveautomation/original/3X/3/f/3f8b6f2b32f3f4b07b382949a0ee9749e74c8390.png)

my goal is to parse out information from a river weather station online.

> **[California Data Exchange Center](https://cdec.water.ca.gov/)**
>
> Water in California is a complex topic. Managing water resources sustainably requires an integrated approach, as water flows across many lines and impacts every Californian. Learn the California Data Exchange Center for better understanding of our...

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 2, 2023, 8:10pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/10 "2023-05-02T20:10:24Z")

</div>

im also assuming i need to put this in gateway startup scripts

---

<div class="post-metadata">

**Author:** ![JordanCClark](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/jordancclark/32/83362_2.png) [@JordanCClark](https://forum.inductiveautomation.com/u/JordanCClark)\
**Post date:** [May 2, 2023, 8:13pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/11 "2023-05-02T20:13:51Z")

</div>

```python
from NewScript import BeautifulSoup

```

Or, rename NewScript...

Also I see an error marker at the bottom. There might be something in the copy/paste that didn't come through.  
 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/f/8/f86d97105fcbe6571f13ac688afd157fb209760d.png)

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 2, 2023, 8:21pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/12 "2023-05-02T20:21:17Z")

</div>

I renamed and published but i still get and also went back to the original name

Traceback (most recent call last):  
File "", line 1, in   
ImportError: No module named NewScript

Edit: also got rid of the red error line.

---

<div class="post-metadata">

**Author:** ![JordanCClark](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/jordancclark/32/83362_2.png) [@JordanCClark](https://forum.inductiveautomation.com/u/JordanCClark)\
**Post date:** [May 2, 2023, 9:55pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/13 "2023-05-02T21:55:04Z")

</div>

Ok, had to spin up a 7.9 VM...

Here's one that will import into 7.9  
[BeautifulSoup\_3.3.2.proj](https://forum.inductiveautomation.com/uploads/short-url/haS8pxRdvxYkqCxv72bcGq9JoWE.proj) (29.8 KB)

Be sure to save after importing.

```python
from project.BeautifulSoup import BeautifulSoup

```

or, if you import it into the global scripts:

```python
from shared.BeautifulSoup import BeautifulSoup

```

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 3, 2023, 2:36pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/14 "2023-05-03T14:36:02Z")

</div>

apologies but im getting this error. im running 7.9.4  
 ![Screenshot 2023-05-03 073523](https://us1.discourse-cdn.com/inductiveautomation/original/3X/e/1/e1b72af5d77f2228c2aee6661ffbf489eb7d9298.png)

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 3, 2023, 2:44pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/15 "2023-05-03T14:44:13Z")

</div>

actually i just changed the rev in the notepad and it imported hopefully it will work i appreciate all the help

---

<div class="post-metadata">

**Author:** ![JordanCClark](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/jordancclark/32/83362_2.png) [@JordanCClark](https://forum.inductiveautomation.com/u/JordanCClark)\
**Post date:** [May 3, 2023, 2:52pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/16 "2023-05-03T14:52:15Z")

</div>

We always recommend upgrading to the latest version. (7.9.21) 😉

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 3, 2023, 3:37pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/17 "2023-05-03T15:37:26Z")

</div>

i dont think i can do this ive tried for the past 16 hours and i cant parse a single html text but beautiful soup works 🙂 at least i think it does.

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [May 3, 2023, 3:50pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/18 "2023-05-03T15:50:14Z")

</div>

> [@Elie\_Khananya](#):
>
> i cant parse a single html text

What's the issue ?

---

<div class="post-metadata">

**Author:** ![Elie\_Khananya](https://avatars.discourse-cdn.com/v4/letter/e/cab0a1/32.png) [@Elie\_Khananya](https://forum.inductiveautomation.com/u/Elie_Khananya)\
**Post date:** [May 3, 2023, 4:04pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/19 "2023-05-03T16:04:01Z")

</div>

im trying to scrap the cfs data on this website [MERCED R NR BRICEBURG #2 (MBG)](https://cdec.water.ca.gov/dynamicapp/QueryF?s=MBG&d=03-May-2023+08:10)

but i just dont know how to write it. i can get the html but idk how to parse it. all the examples online don't really help bc the commands dont work. i was also assuming i need to download pip.

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [May 3, 2023, 4:21pm UTC](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155/20 "2023-05-03T16:21:33Z")

</div>

I can't even open that page ;D

edit: I have some free time (but not a lot), so if you want help parsing html, I'll be there for a little while.

[Next page](https://forum.inductiveautomation.com/t/web-scraper-beautifulsoup/73155.md?page=2)