I am reading in a parquet file and I want to loop over each of the rows and right now I am converting to a pandas dataframe and then using pandas.DataFrame.itertuples to access each row. Is there a way to do this all in pyarrow and not convert to pandas? Just looking at ways to optimize.
One thing I have tried is to convert the pyarrow table to a list of dictionaries that are then looped over, but that is much slower in my case than the conversion to pandas and using itertuples. I was surprised it was slower.
What I mean by that is given pyarrow table t (read from pyarrow.parquet.read_table)
new_list = 
for i in range(t.num_rows):
new_dict = dict()
for k in t.column_names:
new_dict[k] = t[k][i].as_py()
now loop over new_list.
The method pyarrow.Table.to_pydict() would be helpful to make the above more concise but I need it oriented like pandas.DataFrame.to_dict('records').
I get this might not be implemented yet, just asking in case I am missing how to do this natively in arrow.