VATUSA 24 hour time lapse (6/16/2017)

Dhruv Kalra

  • ZMP Staff
  • 431
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #30 on: June 25, 2017, 12:57:04 AM »
Dhruv Kalra
ZMP ATM | Instructor | Grumpy Old Man

Brin Brody

  • VATUSA Staff
  • 275
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #31 on: June 25, 2017, 11:18:44 AM »
Doing one for the Anchorage FNO in August or should I? :)
Brin Brody | VATUSA8
Training Services Manager

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #32 on: June 26, 2017, 11:55:07 AM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #33 on: June 26, 2017, 12:15:53 PM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #34 on: June 26, 2017, 06:44:44 PM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #35 on: June 27, 2017, 01:42:05 AM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

That's a lot of bandwidth I don't want to pay for ;)

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #36 on: June 27, 2017, 07:08:39 PM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

That's a lot of bandwidth I don't want to pay for ;)

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Given the absence of response, I'll let my imagination assume that it's an affirmative response :D

Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #37 on: June 27, 2017, 07:24:06 PM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

That's a lot of bandwidth I don't want to pay for ;)

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Given the absence of response, I'll let my imagination assume that it's an affirmative response :D

My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for.  That's the reason the API isn't public and won't be public.

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #38 on: June 27, 2017, 08:51:26 PM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

That's a lot of bandwidth I don't want to pay for ;)

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Given the absence of response, I'll let my imagination assume that it's an affirmative response :D

My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for.  That's the reason the API isn't public and won't be public.

Are you accumulating the data simply by scraping the status output?

Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #39 on: June 27, 2017, 09:41:33 PM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

That's a lot of bandwidth I don't want to pay for ;)

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Given the absence of response, I'll let my imagination assume that it's an affirmative response :D

My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for.  That's the reason the API isn't public and won't be public.

Are you accumulating the data simply by scraping the status output?

Lots of math and frequent updates.  Since AirCharts serves a mirror of the data feeds, I get to process them more rapidly than the 2-4 minutes other sites are generally restricted to.  There are a number of calculations done and analysis before the position report is logged.

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #40 on: June 28, 2017, 11:50:38 AM »
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?

All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.

That's a lot of bandwidth I don't want to pay for ;)

Unfortunate.

So next question... expose the data via some kind of web services for our consumption? :-)

Given the absence of response, I'll let my imagination assume that it's an affirmative response :D

My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for.  That's the reason the API isn't public and won't be public.

Are you accumulating the data simply by scraping the status output?

Lots of math and frequent updates.  Since AirCharts serves a mirror of the data feeds, I get to process them more rapidly than the 2-4 minutes other sites are generally restricted to.  There are a number of calculations done and analysis before the position report is logged.

Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?

Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #41 on: June 28, 2017, 01:30:58 PM »
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?

No.  There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either.  In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated.  Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans.  Just happened to be one of the benefits having direct access to it.

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #42 on: June 28, 2017, 04:53:27 PM »
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?

No.  There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either.  In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated.  Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans.  Just happened to be one of the benefits having direct access to it.

Interesting.  I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.

Is there any mechanism for individual users to get direct data feed access?  This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page.  I'd almost be willing to host another mirror if it could get me the data I wanted.

Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #43 on: June 28, 2017, 08:04:20 PM »
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?

No.  There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either.  In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated.  Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans.  Just happened to be one of the benefits having direct access to it.

Interesting.  I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.

Is there any mechanism for individual users to get direct data feed access?  This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page.  I'd almost be willing to host another mirror if it could get me the data I wanted.

Negative.  And there are some serious bandwidth draws.. in excess of 300GB per month from my mirror (and I pull from my feeds via the harddisk).  Running a mirror isn't a small thing, really.  In comparison, the VATUSA website pulls between 6 and 7GB per month and Aircharts pulls about 3 and 4GB.

Matthew Kosmoski

  • Members
  • 654
    • View Profile
Re: VATUSA 24 hour time lapse (6/16/2017)
« Reply #44 on: June 29, 2017, 09:01:42 AM »
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?

No.  There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either.  In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated.  Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans.  Just happened to be one of the benefits having direct access to it.

Interesting.  I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.

Is there any mechanism for individual users to get direct data feed access?  This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page.  I'd almost be willing to host another mirror if it could get me the data I wanted.

Negative.  And there are some serious bandwidth draws.. in excess of 300GB per month from my mirror (and I pull from my feeds via the harddisk).  Running a mirror isn't a small thing, really.  In comparison, the VATUSA website pulls between 6 and 7GB per month and Aircharts pulls about 3 and 4GB.

Matter of perspective, I suppose.  300GB/mo is small potatoes these days.