Kindling - a collection of tools to help with Ignition

Ignition
21

You taking feature requests? I often find a time-based accounting graph of log events after optionally filtering them useful. Just a simple count of how frequently messages occur over each span of time.

I already have a Python script which ingests wrapper logs from stdin, looks at the timestamp on each line, sorts them, counts how many occur in each 5 minute bucket, and outputs a CSV of bucket timestamps and counts. I usually then open that up in Excel and graph it. This is neat since I can grep to filter like this:

grep ForwardTransaction wrapper.log | python time_histogram.py > time.csv

But it would be awesome if kindling could make a graph like that with one click from an already filtered log view. :slight_smile:

(If anyone wants my Python script I can share it, I didn’t want to hijack this thread…)

22

Just used it today for an Ignition crash, very useful. Can we disable the file association that the installer does with *.log files? I still like those to open by default in VSCode.

23

Hm, can you show a screenshot of what the end result looks like? It would definitely be possible (I'd just need to pull in JFreeChart, probably), but I'm not sure I have the right idea of what the end product you're looking for is.

Yeah, that would definitely be possible. I'll probably remove all the automatic file associations from the installer and let anyone set them up themselves as they want to.

24

So the intermediary output from my script is a CSV like this:

2022-02-15 15:15:00,0
2022-02-15 15:20:00,0
2022-02-15 15:25:00,1
2022-02-15 15:30:00,3
2022-02-15 15:35:00,2
2022-02-15 15:40:00,1
2022-02-15 15:45:00,2

For each 5 minute interval there is a timestamp of when it starts and then a count of how many log entries were in that time period (after filtering).

The graphical output would ideally be like a time trend chart from Perspective/Vision. I've been too lazy to figure out how to get datetimes nicely on my X axis and instead groom them to Unix timestamps, but my graphs look like this from my spreadsheet software:

time_hist_sample
time_hist_sample832×540 25.8 KB

From this I can notice spikes, maybe notice patterns, and find interesting time periods to look at the log more closely. Like in this one, I might be curious why I had over 90 log messages in 5 minutes roughly around time 1644675000 (February 12, 2022 2:10:00 PM). (I use https://www.epochconverter.com/ quite a lot...)

2 Likes
25

Hm, I like that idea a lot. I’ll have to think about how to implement it, but I’ll keep thinking about it.
I made a formal issue, if you’ve got a Github account feel free to add any thoughts: Graph-view (log events / time slice) · Issue #44 · paul-griffith/kindling · GitHub

1 Like
26

Hey Paul, I have a few requests:

  1. Are you able to add an option to remove the timestamp from the log comment? This makes it easier to see repeating errors.
    image

  2. It looks like you’re converting the local timestamp into GMT?

E.g.
from the raw wrapper:

INFO   | jvm 1    | 2022/03/08 19:19:44 | W [g.D.Route                     ] [08:49:44]: Error handling route. route=/fetch/:project/:uuid, route-group=perspective, route-path=/fetch/:project/:uuid

where 19:19:14 is the local time (+10:30) and 8:49:44 is GMT time

from Kindling:

image
image1044×22 973 Bytes

Would you be able to add an option to select between GMT and local?

  1. Would you be able to add an “export to CSV” option or to Excel? Just an FYI, in the script I wrote to parse wrappers, I replaced all double quotes with single quotes so that I could wrap each row cell in double quotes, unless of course you use an obscure delimiter. I assume if you export to Excel directly though you wouldn’t need to do that.
27

Hmm... Maybe. It should be possible to extract it safely. I don't know offhand whether the timezone ever differs from the actual wrapper log time.

Yeah, for lack of a better option. I can put in a timezone selector (that defaults to local); that's probably the easiest thing. https://github.com/paul-griffith/kindling/issues/47

https://github.com/paul-griffith/kindling/issues/8

1 Like
28

image
image

Haha nice. My reasoning for this is so that we can more easily track using Excel which issues we've resolved and which we're still looking into, and add additional columns etc.

The two timestamps are always different in my wrappers, with the main one that includes the date being local and the one in the log comment being GMT which only has the time part (which is a bit odd, especially with me being +9/10:30 different, the local date is very likely to be different to the GMT date, assuming of course that the gateway timezone is set to local time and not GMT.)

29

7 posts were split to a new topic: Windows Problems

30

v0.2.0:
Adds log ‘density’ display to scrollbar in log viewer (thanks for the idea, @justin.brzozoski)
This is an experimental/incubating feature. Currently, logs are aggregated into 2-minute buckets and rendered in the scrollbar:

image
image1283×801 141 KB

I also changed the wrapper log parsing to remove the extra timestamps (@nminchin). And added a timezone dropdown at the top of the panel.

4 Likes
31

Thanks for adding timezone :slight_smile: and that density graph looks sweeeet :heart_eyes:

32

By dumb luck, the recent log I grabbed off one of our gateways to try out the new Kindling had one huge spike in density. I didn’t know that going in and was kinda confused why the graph was mostly empty when I opened it.

kindling_sample
kindling_sample86×539 2.02 KB

That one faint spike 70-80% of the way down just dwarfs everything else:
excel_sample

And that spike was from a mostly harmless bug in our code I’m off to go fix now… :slight_smile:

33

I've noticed similar things a few times; I might try to get fancier and do a log scale horizontally or cap the absolute magnitude. What I'm doing is basically manually rendering a graph and then scaling it down to fit inside the scrollbar, so when there's lots of detail you can get some very faint lines.
I'll probably add time-slice selection in the column settings in the next release - maybe 1, 2, 5, 10 minutes or something like that.

34

I suspect you also know this, but it’s difficult to tell where the log lines currently in the view exist on the time graph. The scrollbar is linear with number of lines, the time graph is linear with time, and they don’t align. I briefly (instinctively) tried to move the scroll bar on top of the spike, then realized that wasn’t going to work. I’m not sure what simple solutions exist for this issue. Maybe just highlight where the current viewable log lines are on the time graph in a second color? But that would require updating/refreshing your scaled graphic, so that might be more processing than it’s worth.

I don’t know what the good solution is here. Something to ponder for now.

I like Kindling a lot. Thank you for putting it together.

1 Like
35

Some tools replace the scrollbar entirely with the mini map:
image

Some go side-by-side:
image

In each case the map has its own position indicator (not shown in first screenshot).

1 Like
36

@PGriffith I couldn’t run it on windows

37

Can you provide any more information? The latest published version (0.2.1) works fine on my windows laptop. The tool I’m using, JDeploy, should automatically download an appropriate JDK for you.

38

0.2.2 is out now - a minor update to tweak the density display and fix the painfully-slow scroll speed of the logs view.

39

Hi @PGriffith is there a way to use Kindling to help us understand quarantine.xml files?
I often find things have been quarantined, but opening the file doesn’t help whatsoever as it is all encrypted. I just want to know what table in the database it is failing with!
Ideally also why it is failing…

Or do i need to get a different file for this analysis?

1 Like
40

The quarantine file is not generally very helpful (it’s missing too much information) but if you zip up the entire folder in the datacache directory in the install directory, you can open that zip using the ‘S+F Cache Zip File’ tool:
image

That should hopefully give you some insight into what’s going wrong.

2 Likes