drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdel Hakim Deneche <adene...@maprtech.com>
Subject Re: strange build problem with generated sources
Date Fri, 05 Feb 2016 17:41:02 GMT
Do you also need to add forceBits to the template FixedValueVectors.java ?

Can you copy the line from VarLengthColumnReaders .java where you hit a
compile error ?


On Fri, Feb 5, 2016 at 9:26 AM, Dave Oshinsky <doshinsky@commvault.com>
wrote:

> Hi Abdel,
> Thanks for reply.  I added it to
> exec/vector/target/codegen/templates/NullableValueVectors.java.  I then see
> it propagated by Freemarker to generated sources, as shown below
> signature.  In Eclipse, I'm able to navigate from where I'm calling
> forceBits() to e.g. NullableDecimal28SparseVector.java, and I see
> forceBits() implemented there.  However, Eclipse has the dreaded "red X" in
> VarLengthColumnReaders.java indicating an error, and the build also fails.
> Weird,  huh?
>
> I do have to say, this whole "generated sources" business is a pain in the
> you know where  ;-)  In all seriousness, I would implement the whole
> decimal business MUCH more simply if writing it from scratch.  Why the
> gazillion special cases for different widths (all with separate generated
> sources), for nullable and not, sparse or dense?  Just treat them all the
> same, at least from where I'm sitting with a Parquet-centric perspective.
> And the memory usage would be MUCH lower by treating decimals ALWAYS as
> variable width.  Most actual decimal numbers are way smaller than the
> precision would indicate, in many of my test cases.
>
> Dave Oshinsky
>
> $ find . -name "*.java" | xargs grep forceBits
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
>     nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
>     nullableDecimal38Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
>     nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
>     nullableDecimal38Vector.forceBits(index, 1);
> ./exec/vector/src/main/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/classes/codegen/templates/NullableValueVectors.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBigIntVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBitVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDateVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal18Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28DenseVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28SparseVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38DenseVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38SparseVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal9Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat4Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat8Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalDayVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalYearVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableSmallIntVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeStampVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTinyIntVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt1Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt2Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt4Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt8Vector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVar16CharVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarBinaryVector.java:
>   public void forceBits(int index, int value){  // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarCharVector.java:
>   public void forceBits(int index, int value){  // DAO
>
> -----Original Message-----
> From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
> Sent: Friday, February 05, 2016 12:10 PM
> To: dev@drill.apache.org
> Subject: Re: strange build problem with generated sources
>
> Hey Dave,
>
> Which file did you add the "forceBits()" method to ?
>
> On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky <doshinsky@commvault.com>
> wrote:
>
> > Hi Drill-ers,
> > I am looking into fixing JIRA
> > https://issues.apache.org/jira/browse/DRILL-4184.  I've encountered a
> > number of strange build problems along the way with my drill 1.4
> > snapshot, including inability to rebuild after running "mvn clean", no
> > matter what I try.  So, I'm building from scratch for the second time,
> > at least.  The latest problem really has me stumped at the moment.  I
> > added a "forceBits(int,int)" method that I see in generated source
> > file NullableDecimal28SparseVector.java (and
> > NullableDecimal38SparseVector.java), but somehow this doesn't get
> > compiled properly into the *.class and my build keeps failing as if
> > the new forceBits method isn't there:
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
> > (default-compile) on project drill-java-exec: Compilation failure:
> > Compilation failure:
> > [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo
> > lumnReaders.java:[108,29]
> > error: cannot find symbol
> > [ERROR] symbol:   method forceBits(int,int)
> > [ERROR] location: variable nullableDecimal28Vector of type
> > NullableDecimal28SparseVector [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo
> > lumnReaders.java:[179,29]
> > error: cannot find symbol
> >
> > Can anyone suggest how to fix this without starting over from scratch
> > in a new build node (again)?  Any advice would be greatly appreciated.
> >
> > I will send a separate email eventually regarding the design of my
> > fix, which I know is only a short-term solution to the problem of
> > handling variable width decimal fields in Parquet files.  To make a
> > long story short, all the decimal vectors are fixed width vectors,
> > which don't have the ability to "remember" varying sizes from one
> > decimal field to the next.  I've hacked up something to "remember" the
> > varying field sizes (BigDecimal array sizes) in
> > NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the
> > decimal vectors.  This seems to work, though it's admittedly ugly.
> > However, I ran into a problem with nullable varying width decimal
> > columns where the "isSet" always returns 0, as if the column is null,
> > when it is not, and the sparse decimal data is present in the vector
> > (but Drill won't send the decimal value, because it thinks it's null).
> > Hence the "forceBits" hack to try to work around this.  It seemed like
> > I was close to running a successful Drill query on the varying width
> > decimal Parquet data, but alas, I ran into (another) build problem.  I
> > do have a LOT of questions as to why the decimal stuff was designed the
> way it is, but that's for another email....
> >
> > Thanks,
> > Dave Oshinsky
> >
> >
> >
> >
> >
> > ***************************Legal Disclaimer***************************
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply
> > email and delete the message. Thank you."
> > **********************************************************************
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
>



-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message